Introduction

This example demonstrates how to compute the probability of success of a clinical trial by extending East Horizon’s single-endpoint framework to handle dual endpoints using custom R scripts for the Response (Patient Simulation) and the Analysis integration points. This example uses Progression Free Survival and Overall Survival as endpoints, however, this could be extended to other types of endpoints by modifying the R code.

Why do we need R Integration for this example?

To compute the Probability of Success of a trial, users need to simulate patient outcomes by sampling the true rates from prior distributions. This is something that East Horizon cannot handle yet with its current response generation algorithms, so we must integrate a custom R file for the simulation to do so.

When a trial involves multiple success criteria, users must change how “Success” is defined from a traditional success analysis with only one endpoint. East Horizon cannot handle multiple success criteria for a Winning Condition yet, so we must integrate a custom R file for the simulation to take into account changing success criteria from the traditional “statistically significant” criteria.

What do the R functions do?

In the R directory of this example you will find the following R files:

  • Simulate2EndpointTTEWithMultiState.R

    This R file contains a function that executes the response data generation for both PFS and OS endpoints. Due to the typically assumed correlation between PFS and OS, the timing of the events for each simulated patient must take into account that these endpoints are not completely independent of each other. This function therefore handles this concern by using a multi-state model that takes into consideration:

    1. The rate at which events occur from trial start to a progression event,
    2. The rate at which events occur from trial start to a death, and
    3. The probability that death takes place before a progression event.

    As there may be uncertainty around each of these assumed parameters, the function simulates each of them based on prior distributions rather than single values for each. Users must provide input assumptions that parametrize the prior distributions that will be sampled from to simulate each patient’s response times.

    For more information on this function, see the Response (Patient Simulation) Integration Point section below.

  • AnalyzePFSAndOS.R

    This R file contains a function that executes the interim and final analyses of each simulated trial. The function takes into account that both PFS and OS event time stamps are being generated. The function computes the probability of success of the trial, where success is defined as:

    1. Statistical significance for PFS: PFS data and frequentist analysis using a Logrank test statistic proves that the difference between the control and treatment arms is proven to be statistically significant.
    2. Positive trend observed for OS: OS data shows a positive trend in the difference between the control and treatment arms – the positive trend is defined by a user-specified threshold for the Hazard Ratio observation.

    The success criteria can be changed to compute the probability of success for varying success criteria. The lines in the code that need to be modified are line 170 for interim analyses and line 193 for final analysis.

    For more information on this function, see the Analysis Integration Point section below.

Combining it all together…

The figure below illustrates where this example fits within the R integration points of Cytel products, accompanied by flowcharts outlining the general steps performed by the R code.

By combining the above R functions with the East Horizon native inputs, users are able to simulate and compute the probability of success of each simulated trial. Users will also continue to benefit from East Horizon’s output visualizations – with the caveat that the Probability of Success metric will be labeled as “Power” in the native outputs of East Horizon.

Step-by-Step Instructions

Before starting, make sure you have the required tools and files.

  1. East Horizon
  2. Download R Files from our public Github repo: Simulate2EndpointTTEWithMultiState.R and AnalyzePFSAndOS.R.

New Project Page

  1. On East Horizon, create a new project with a single endpoint.

New Input Set Page

  1. Navigate to the Inputs tab, and create a new input set using the Explore task.

Design Page

  1. Click on the input set you just created, then select “User Specified – R” in the Test Statistic field.

  1. Click the “+” icon to open the R Integration pop-up window.

  1. Click on “Select File” and then on “Continue”.

  1. Select the file “AnalyzePFSAndOS.R”.

  1. Check that the correct file has been imported and the correct Function Name has been specified by the system.

  1. Note that the User Parameter variables have been automatically pulled from the R function that was imported. Specify the values for each of these variables.

These variables are the thresholds for the Hazard Ratio for Overall Survival to be considered “TRENDING” for the trial’s second success criteria. Users must specify a threshold for the interim analysis, if using a group sequential design, and always for the final analysis. See the Analysis Integration Point section below for more information about these variables and example values.

Note: We typically make success criteria more CONSERVATIVE (i.e. making it more difficult to declare a “win”) on a trending OS at the interim analysis compared to the final analysis because you want to be “more sure” of your decision when you have less evidence/data available at the time of decision.

  1. Click on the “Save” button to exit the R Integration details window.

Response Page

  1. Start with identifying:
  1. Median time to PFS for Control arm & Variance.
  2. Median time to PFS for Experimental arm (this can be computed using above parameter and an assumed Hazard Ratio) & Variance.
  3. Median time to OS for Control arm & Variance.
  4. Median time to OS for Experimental arm (this can be computed using above parameter and an assumed Hazard Ratio) & Variance.
  5. Probability of Death event Happening BEFORE Progression event for Control arm.
  6. Probability of Death event Happening BEFORE Progression event for Treatment arm.

See the Response (Patient Simulation) Integration Point section below for more information about these variables and example values.

  1. Use the Parameter Solving Tool below to convert Median Survival Time & Variance parameters into Shape & Rate parameters.
  1. Navigate to the Response page, and then select “User Specified – R” in the Distribution field.

  1. Click on the “+” icon to open the R Integration pop-up window.

  1. Click on “Select File” and then on “Continue”.

  1. Select the file “Simulate2EndpointTTEWithMultiState.R”.

  1. Check that the correct file has been imported and the correct Function Name has been specified by the system.

  1. Note that the User Parameter variables have been automatically pulled from the R function that was imported. Delete the first six variables from the list, as we are not using them right now. As the Delete feature is not available yet, you can specify a value of 0 for each parameter instead.

  1. Specify the values for each of the required User Parameter variables for the function.

See the Response (Patient Simulation) Integration Point section below for more information on these variables and example values.

  1. Click on the “Save” button to exit the R Integration details window.

  1. Navigate to the “Dropout Rate” tab within the Response Card to include dropout information if needed.

Enrollment Page

  1. Navigate to the Enrollment page, and specify the average number of subjects enrolled per time unit (e.g. default is per month).

Simulation Setup Page

  1. Specify the number of simulation runs as needed.

  1. Check the checkbox to save the simulation data for all simulated models.

  1. Click the “Save & Simulate” button.

  1. Confirm by clicking on “Simulate” in the pop-up window, and wait for the simulation runs to finish.

Results Page

  1. Check whether the simulation failed or was completed by looking at the Status column.

  1. If the simulation failed, open the Log window to see if there are any helpful error messages.
  1. Click on the “?” icon in the top right corner of your screen.

  1. Click on “Log”.

  1. Identify any errors that appear. For example:

  1. If the simulation has completed, click on the Result name. If you have multiple scenarios in your simulation, you will be prompted to label at least one scenario from the list. Alternatively, you may also skip this step by selecting the “Skip” button in the bottom left corner.

  1. The Explore page of the results appears. Hover over each cell in the heat map to see a summary of the outputs. Because of our custom R scripts, “Power” is actually now equivalent to “Probability of Success”, where success means that PFS is statistically significant and OS is trending.

See the Results section below for more information.

  1. To visualize the Probability of Success at an interim look, click on the cell in the heat map to open the Detailed Model Output Card. Scroll to the bottom and open the “Simulation Boundaries and Boundary Crossing Probabilities” Table.

  1. If you need to be reminded which threshold for the OS success criteria was used, scroll up to the top of the Detailed Output page and find details of your inputs in the “Summary” card.

  1. Compute the Probability of Success at each analysis by identifying the number of times the trial stopped for Efficacy at a given analysis over the total number of simulations. For example:

  • Probability of Success at interim: 4055/10000 = 40.55%
  • Probability of Success at the final analysis conditional on an unsuccessful interim: 3191/5945 = 53.68%
  • Probability of Success by the final analysis (either at IA1 or FA): 7246/10000 = 72.46%

Technical Information and Example Values

Response (Patient Simulation) Integration Point

We are using East Horizon’s single-endpoint framework, which we customize to support dual endpoints through the Response Integration Point via our R script linked above. The two endpoints of interest are:

  • Progression-Free Survival (PFS): The duration during which a patient lives with a disease without experiencing further progression.
  • Overall Survival (OS): The total time elapsed from treatment until death.

We use a multi-state model to simulate event times for each patient in every simulated trial. This model captures the relationship between PFS and OS by generating both outcomes together, rather than independently. Information generated from this simulation will be used later for the Analysis Integration Point.

The diagram above illustrates the multistate model for survival analysis, with three states:

  1. State 0: Initial, no progression.
  2. State 1: Progression, event occurs that makes the disease progress.
  3. State 2: Death.

There are also three transitions between states:

  1. \(\alpha_{01}\): Transition rate from Initial to Progression.
  2. \(\alpha_{02}\): Transition rate from Initial to Death.
  3. \(\alpha_{12}\): Transition rate from Progression to Death.

From Meller et al. [1], the probability of progression before death is:

\[ q = \frac{\alpha_{01}}{\alpha_{01} + \alpha_{02}} \]

Where \(q = 1 - p\), with \(p\) being the probability of Death before Progression.

Using this relationship:

\[ \alpha_{02} = \frac{\alpha_{01} \cdot p}{q} \]

Or equivalently:

\[ \alpha_{01} = \frac{\alpha_{02} \cdot q}{p} \]

Progression-Free Survival (PFS) is treated as the minimum of two competing times:

  • \(X_1 \sim \text{Exp}(\alpha_{01})\): Time from Initial to Progression.
  • \(X_2 \sim \text{Exp}(\alpha_{02})\): Time from Initial to Death without Progression.

Using the property of the minimum of two exponential random variables [2]:

\[ Z = \min(X_1, X_2) \sim \text{Exp}(\alpha_{01} + \alpha_{02}) \]

The median of an exponential distribution is given by:

\[ M = \frac{\log(2)}{\alpha_{01} + \alpha_{02}} \]

Substituting \(\alpha_{01}\) in terms of \(\alpha_{02}\):

\[ M = \frac{1}{\alpha_{01} + \alpha_{02}} = \frac{1}{\frac{\alpha_{02} \cdot q}{p} + \alpha_{02}} = \frac{1}{\alpha_{02} \cdot \left(\frac{q}{p} + 1\right)} \]

Solving for \(\alpha_{02}\):

\[ \alpha_{02} = \frac{\log(2)}{M \cdot \left(\frac{q}{p} + 1\right)} \]

Once \(\alpha_{02}\) is calculated, \(\alpha_{01}\) can be derived using:

\[ \alpha_{01} = \frac{\alpha_{02} \cdot q}{p} \]

Overall Survival (OS) accounts for both pathways:

  1. Directly Initial to Death without Progression:
    • Occurs with probability \(p\).
    • Survival time is \(OS = X_2\) (time to death without progression).
  2. Initial to Progression to Death:
    • Occurs with probability \(q\).
    • Survival time is \(OS = X_1 + X_3\), where:
      • \(X_1 \sim \text{Exp}(\alpha_{01})\) (time to progression),
      • \(X_3 \sim \text{Exp}(\alpha_{12})\) (time from progression to death).

The median OS (\(M_{OS}\)) is approximated as a weighted combination of these two pathways:

\[ M_{OS} = \text{Median}(X_2 \cdot p + (X_1 + X_3) \cdot q) = p \cdot \frac{\log(2)}{\alpha_{02}} + q \cdot (\frac{\log(2)}{\alpha_{01}} + \frac{\log(2)}{\alpha_{12}})\]

To compute \(\alpha_{12}\), the model numerically solves for the rate using uniroot, ensuring consistency with the user-provided median OS.

Sources

  • [1] Joint Modeling of progression-free and overall survival and computation of correlation measures, Meller M, Beyersmann J, Rufibach K, 2018
  • [2] Purdue University, minimum_two_exponentials.pdf

Example Using Median Times

One option is to directly input the median times and probabilities of death before progression into the script as user parameters. Refer to the table below for the definitions and example values of these user-defined parameters.

User parameter Definition Value
dMedianPFS0 Median time to PFS event for control group 12
dMedianPFS1 Median time to PFS event for treatment group 18
dMedianOS0 Median time to OS event for control group 18
dMedianOS1 Median time to OS event for treatment group 27
dProbOfDeathBeforeProgression0 Probability of death before PFS for control group 0.2
dProbOfDeathBeforeProgression1 Probability of death before PFS for treatment group 0.2

Here, the median times are in months and the hazard ratio for both endpoints (PFS & OS) is equal to \(\frac{12}{18} = \frac{18}{27} = 0.6667\). The probability of death before progression is 20% for both control and treatment arms.

Example Using Prior Distributions

Another option is to customize how patient data is simulated by building a more realistic model for both PFS and OS outcomes using prior distributions instead of directly using median times and probabilities. Using prior distributions allows us to account for uncertainty around the true treatment effect, which enables users to identify the Probability of Success of a trial rather than statistical Power. Here’s how the event data is generated:

  • Time to PFS event is sampled from a Gamma distribution.
  • Time to OS event is also sampled from a Gamma distribution.
  • The probability that the OS event happens before the PFS event is sampled from a Beta distribution.

All three parameters are treated as random variables, sampled from prior distributions, allowing each simulated trial to reflect a range of possible real-world outcomes.

Refer to the table below for the definitions of the user-defined parameters used in this example.

User Parameter Definition
dMedianPFS0PriorShape Shape parameter for the median time to PFS event for control group
dMedianPFS0PriorRate Rate parameter for the median time to PFS event for control group
dMedianPFS1PriorShape Shape parameter for the median time to PFS event for treatment group
dMedianPFS1PriorRate Rate parameter for the median time to PFS event for treatment group
dMedianOS0PriorShape Shape parameter for the median time to OS event for control group
dMedianOS0PriorRate Rate parameter for the median time to OS event for control group
dMedianOS1PriorShape Shape parameter for the median time to OS event for treatment group
dMedianOS1PriorRate Rate parameter for the median time to OS event for treatment group
dProbOfDeathBeforeProgression0Param1 Alpha parameter for probability of death before progression for control group
dProbOfDeathBeforeProgression0Param2 Beta parameter for probability of death before progression for control group
dProbOfDeathBeforeProgression1Param1 Alpha parameter for probability of death before progression for treatment group
dProbOfDeathBeforeProgression1Param2 Beta parameter for probability of death before progression for treatment group

The shape and rate parameters can be calculated from the assumed mean (i.e. median survival time) and its variance. You can use the Parameter Solving tool from the Step-by-Step section above to compute these required parameters.

If you have the scale parameter for your assumption instead of the rate parameter, you can convert from one to the other using the formula \(Rate = \frac{1}{Scale}\).

Scenario 1: Alternative Hypothesis & Equal Variance

In this first example scenario, we want:

  • A variance of 10 for both endpoints (PFS and OS) and both arms (control and treatment).
  • For the control arm:
    • A mean time to PFS event of 12 months.
    • A mean time to OS event of 18 months.
  • For the treatment arm:
    • A mean time to PFS event of 18 months.
    • A mean time to OS event of 27 months.
  • A probability of death before progression of 20% for both arms.

Using the tool above, we get:

  • For \(mean = 12\), \(variance = 10\) \(\rightarrow\) \(shape = 14.4\), \(rate = 1.2\)
  • For \(mean = 18\), \(variance = 10\) \(\rightarrow\) \(shape = 32.4\), \(rate = 1.8\)
  • For \(mean = 27\), \(variance = 10\) \(\rightarrow\) \(shape = 72.9\), \(rate = 2.7\)

Refer to the table below for the values of all user-defined parameters used in this example.

User parameter Value
dMedianPFS0PriorShape 14.4
dMedianPFS0PriorRate 1.2
dMedianPFS1PriorShape 32.4
dMedianPFS1PriorRate 1.8
dMedianOS0PriorShape 32.4
dMedianOS0PriorRate 1.8
dMedianOS1PriorShape 72.9
dMedianOS1PriorRate 2.7
dProbOfDeathBeforeProgression0Param1 20
dProbOfDeathBeforeProgression0Param2 80
dProbOfDeathBeforeProgression1Param1 20
dProbOfDeathBeforeProgression1Param2 80

Scenario 2: Alternative Hypothesis & Higher Variance

In this second example scenario, we want:

  • For the control arm:
    • A variance of 10 for both endpoints (PFS and OS).
    • A mean time to PFS event of 12 months.
    • A mean time to OS event of 18 months.
  • For the treatment arm:
    • A variance of 20 for both endpoints (PFS and OS).
    • A mean time to PFS event of 18 months.
    • A mean time to OS event of 27 months.
  • A probability of death before progression of 20% for both arms.

Using the tool above, we get:

  • For \(mean = 12\), \(variance = 10\) \(\rightarrow\) \(shape = 14.4\), \(rate = 1.2\)
  • For \(mean = 18\), \(variance = 10\) \(\rightarrow\) \(shape = 32.4\), \(rate = 1.8\)
  • For \(mean = 18\), \(variance = 20\) \(\rightarrow\) \(shape = 16.2\), \(rate = 0.9\)
  • For \(mean = 27\), \(variance = 20\) \(\rightarrow\) \(shape = 36.45\), \(rate = 1.35\)

Refer to the table below for the values of all user-defined parameters used in this example.

User parameter Value
dMedianPFS0PriorShape 14.4
dMedianPFS0PriorRate 1.2
dMedianPFS1PriorShape 16.2
dMedianPFS1PriorRate 0.9
dMedianOS0PriorShape 32.4
dMedianOS0PriorRate 1.8
dMedianOS1PriorShape 36.45
dMedianOS1PriorRate 1.35
dProbOfDeathBeforeProgression0Param1 20
dProbOfDeathBeforeProgression0Param2 80
dProbOfDeathBeforeProgression1Param1 20
dProbOfDeathBeforeProgression1Param2 80

Scenario 3: Null Hypothesis & Equal Variance

In this third example scenario, we want:

  • A variance of 10 for both endpoints (PFS and OS) and both arms (control and treatment).
  • For the control arm:
    • A mean time to PFS event of 12 months.
    • A mean time to OS event of 18 months.
  • For the treatment arm:
    • A mean time to PFS event of 12 months.
    • A mean time to OS event of 18 months.
  • A probability of death before progression of 20% for both arms.

Using the tool above, we get:

  • For \(mean = 12\), \(variance = 10\) \(\rightarrow\) \(shape = 14.4\), \(rate = 1.2\)
  • For \(mean = 18\), \(variance = 10\) \(\rightarrow\) \(shape = 32.4\), \(rate = 1.8\)

Refer to the table below for the values of all user-defined parameters used in this example.

User parameter Value
dMedianPFS0PriorShape 14.4
dMedianPFS0PriorRate 1.2
dMedianPFS1PriorShape 14.4
dMedianPFS1PriorRate 1.2
dMedianOS0PriorShape 32.4
dMedianOS0PriorRate 1.8
dMedianOS1PriorShape 32.4
dMedianOS1PriorRate 1.8
dProbOfDeathBeforeProgression0Param1 20
dProbOfDeathBeforeProgression0Param2 80
dProbOfDeathBeforeProgression1Param1 20
dProbOfDeathBeforeProgression1Param2 80

Scenario 4: Null Hypothesis & Higher Variance

In this final example scenario, we want:

  • For the control arm:
    • A variance of 10 for both endpoints (PFS and OS)
    • A mean time to PFS event of 12 months.
    • A mean time to OS event of 18 months.
  • For the treatment arm:
    • A variance of 20 for both endpoints (PFS and OS)
    • A mean time to PFS event of 12 months.
    • A mean time to OS event of 18 months.
  • A probability of death before progression of 20% for both arms.

Using the tool above, we get:

  • For \(mean = 12\), \(variance = 10\) \(\rightarrow\) \(shape = 14.4\), \(rate = 1.2\)
  • For \(mean = 18\), \(variance = 10\) \(\rightarrow\) \(shape = 32.4\), \(rate = 1.8\)
  • For \(mean = 12\), \(variance = 20\) \(\rightarrow\) \(shape = 7.2\), \(rate = 0.6\)
  • For \(mean = 18\), \(variance = 20\) \(\rightarrow\) \(shape = 16.2\), \(rate = 0.9\)

Refer to the table below for the values of all user-defined parameters used in this example.

User parameter Value
dMedianPFS0PriorShape 14.4
dMedianPFS0PriorRate 1.2
dMedianPFS1PriorShape 7.2
dMedianPFS1PriorRate 0.6
dMedianOS0PriorShape 32.4
dMedianOS0PriorRate 1.8
dMedianOS1PriorShape 16.2
dMedianOS1PriorRate 0.9
dProbOfDeathBeforeProgression0Param1 20
dProbOfDeathBeforeProgression0Param2 80
dProbOfDeathBeforeProgression1Param1 20
dProbOfDeathBeforeProgression1Param2 80

Analysis Integration Point

Using the file above, the Analysis element of East Horizon’s simulation is customized to compute the probability of success (PoS) for the trial, based on dual endpoints: Progression-Free Survival (PFS) and Overall Survival (OS). The file uses information from the simulation (SimData variable) that is generated by the Response element of East Horizon’s simulation. See the Response section above for more information about the PFS and OS endpoints generation.

The criteria for declaring trial success are as follows. Both criteria must be met to declare success:

  • PFS Endpoint: Statistical significance must be achieved by crossing the predefined efficacy boundary (defined by East Horizon, using a frequentist analysis).
  • OS Endpoint: A positive trend must be observed, defined as the OS hazard ratio being below a pre-specified threshold (defined with user parameters).

Note: At an interim analysis, we usually set stricter criteria for a positive trend on OS compared to the final analysis. This is because we have less data early on, so we want to be more confident in any decision we make at that stage. Therefore, the threshold defining a positive trend at the interim is typically lower than the threshold used at the final analysis.

Refer to the table below for the definitions of the user-defined parameters used in this example.

User parameter Definition
HazardRatioCutoffIA OS hazard ratio threshold used for the interim analysis
HazardRatioCutoffFA OS hazard ratio threshold used for the final analysis

Option 1: Fixed Sample Design

The first example option is a fixed sample design with a hazard ratio threshold of 0.9. This option shows that we can still use this script without interim analyses. Refer to the table below for the values of the user-defined parameters used in this option.

User parameter Value
HazardRatioCutoffFA 0.9

Option 2: Group Sequential Design

The second example option is a group sequential design with hazard ratio thresholds of 0.8 (for interim analysis) and 0.9 (for final analysis). Refer to the table below for the values of the user-defined parameters used in this option.

User parameter Value
HazardRatioCutoffIA 0.8
HazardRatioCutoffFA 0.9

Option 3: Low PoS Group Sequential Design

The third example option is a group sequential design with a lower hazard ratio threshold of 0.5 for both interim and final analyses. Refer to the table below for the values of the user-defined parameters used in this option.

User parameter Value
HazardRatioCutoffIA 0.5
HazardRatioCutoffIA 0.5

Option 4: High PoS Group Sequential Design

The fourth example option is a group sequential design with a lower hazard ratio threshold of 0.5 (for interim analysis) and a higher hazard ratio threshold of 1.2 (for final analysis). Refer to the table below for the values of the user-defined parameters used in this option.

User parameter Value
HazardRatioCutoffIA 0.5
HazardRatioCutoffIA 1.2

Results

In the Results section, Power now refers to the probability of success, where success is defined as a statistically significant difference in time to progression-free survival (PFS) between the control and treatment arm as well as a positive trend in overall survival (OS), i.e. the time to overall survival (OS) is longer in magnitude for the patients in the treatment arm compared to those in the control arm. Below is an example of the heatmap that could be generated by East Horizon following the simulation. Each square represents the simulated probability of success for a trial, based on a specific combination of Response scenario (columns) and Analysis option (rows).